{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6f63eab6-9d70-497c-9743-2acd652b7b8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# smoke_test = True\n",
    "\n",
    "import gpytorch\n",
    "import torch\n",
    "\n",
    "from torch.utils import benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "057df92a-445c-4d33-b914-4fb39d9b7c7b",
   "metadata": {},
   "source": [
    "# Kernels with Additive or Product Structure\n",
    "\n",
    "One of the most powerful properties of kernels is their closure under various composition operation.\n",
    "Many important covariance functions can be written as the sum or the product of $m$ component kernels:\n",
    "\n",
    "$$\n",
    "    k_\\mathrm{sum}(\\boldsymbol x, \\boldsymbol x') = \\sum_{i=1}^m k_i(\\boldsymbol x, \\boldsymbol x'), \\qquad\n",
    "    k_\\mathrm{prod}(\\boldsymbol x, \\boldsymbol x') = \\prod_{i=1}^m k_i(\\boldsymbol x, \\boldsymbol x')\n",
    "$$\n",
    "\n",
    "Additive and product kernels are used for a variety of reasons.\n",
    "1. They are often more interpretable, as argued in [Duvenaud et al. (2011)](https://arxiv.org/pdf/1112.4394).\n",
    "2. They can be extremely powerful and expressive, as demonstrated by [Wilson and Adams (2013)](https://proceedings.mlr.press/v28/wilson13.pdf).\n",
    "3. They can be extremely sample efficient for Bayesian optimization, as demonstrated by [Kandasamy et al. (2015)](https://arxiv.org/pdf/1503.01673) and [Gardner et al. (2017)](https://proceedings.mlr.press/v54/gardner17a/gardner17a.pdf).\n",
    "\n",
    "We will discuss various ways to perform additive and product compositions of kernels in GPyTorch.\n",
    "The simplest mechanism is to add/multiply the kernel objects together, or add/multiply their outputs.\n",
    "However, there are more complex but **far more efficient ways** for adding/multiplying kernels with similar functional forms, which will enable significant parallelism especially on GPUs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "610e82ae-227b-4c0d-b817-9cd6baa73d92",
   "metadata": {},
   "source": [
    "## Simple Sums and Products\n",
    "\n",
    "As an example, consider the [spectral mixture kernel](https://docs.gpytorch.ai/en/stable/kernels.html#spectralmixturekernel) with two components on a univariate input.\n",
    "If we remove the scaling components, it can be implemented as:\n",
    "\n",
    "$$\n",
    "    k_\\mathrm{SM}(x, x') =\n",
    "    k_\\mathrm{RBF}(x, x', \\ell_1) k_\\mathrm{cos}(x, x'; \\omega_1) +\n",
    "    k_\\mathrm{RBF}(x, x', \\ell_2) k_\\mathrm{cos}(x, x'; \\omega_2),\n",
    "$$\n",
    "\n",
    "where $\\ell_1, \\ell_2, \\omega_1, \\omega_2$ are hyperparameters. We can naively implement this kernel in two ways..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e060386d-5192-4d03-9002-2f4169be7507",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Toy data\n",
    "X = torch.randn(10, 1)\n",
    "\n",
    "# Base kernels\n",
    "rbf_kernel_1 = gpytorch.kernels.RBFKernel()\n",
    "cos_kernel_1 = gpytorch.kernels.CosineKernel()\n",
    "rbf_kernel_2 = gpytorch.kernels.RBFKernel()\n",
    "cos_kernel_2 = gpytorch.kernels.CosineKernel()\n",
    "\n",
    "# Implementation 1:\n",
    "spectral_mixture_kernel = (rbf_kernel_1 * cos_kernel_1) + (rbf_kernel_2 * cos_kernel_2)\n",
    "covar = spectral_mixture_kernel(X)\n",
    "\n",
    "# Implementation 2:\n",
    "covar = rbf_kernel_1(X) * cos_kernel_1(X) + rbf_kernel_2(X) * cos_kernel_2(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddf6ee94-f7eb-4bb2-935a-7a76787fa596",
   "metadata": {},
   "source": [
    "Implementation 1 constructs a `spectral_mixture_kernel` object by applying `+` and `*` directly to the component kernel objects.\n",
    "Implementation 2 constructrs the resulting covariance matrix by applying `+` and `*` to the outputs of the component kernels.\n",
    "Both implementations are equivalent (the `spectral_mixture_kernel` object created by Implementation 1 essentially performs Implementation 2) under the hood.\n",
    "\n",
    "(Of course, neither implementation should be used in practice for the spectral mixture kernel. The built-in [SpectralMixtureKernel](https://docs.gpytorch.ai/en/stable/kernels.html#spectralmixturekernel) class is far more efficient.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfbc6181-752d-4905-8e08-df27fb5a93d5",
   "metadata": {},
   "source": [
    "## Efficient Parallel Implementations of Additive Structure or Product Structure Kernels   \n",
    "\n",
    "Above we considered the sum and products of kernels with different functional forms.\n",
    "However, often we are considering the sum/product over kernels with \n",
    "The above example is simple to read, but quite slow in practice.\n",
    "Under the hood, each of the kernels (and their compositions) are computed sequentially.\n",
    "GPyTorch will compute the first cosine kernel, followed by the first RBF kernel, followed by their product, and so on.\n",
    "\n",
    "When the component kernels have the same function form,\n",
    "we can get massive efficieny gains by exploiting parallelism.\n",
    "We combine all of the component kernels into a **batch kernel**\n",
    "so that each component kernel can be computed simultaneously.\n",
    "We then compute the `sum` or `prod` over the batch dimension.\n",
    "This strategy will yield significant speedups especially on the GPU."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "731fc597-ccbc-441a-830d-0b8d4efe100d",
   "metadata": {},
   "source": [
    "### Example #1: Efficient Summations of Univariate Kernels\n",
    "\n",
    "As an example, let's assume that we have $d$-dimensional input data $\\boldsymbol x, \\boldsymbol x' \\in \\mathbb R^d$.\n",
    "We can define an *additive kernel* that is the sum of $d$ univariate RBF kernels, each of which acts on a single dimension of $\\boldsymbol x$ and $\\boldsymbol x'$.\n",
    "\n",
    "$$\n",
    "    k_\\mathrm{additive}(\\boldsymbol x, \\boldsymbol x') =  \\prod_{i=1}^d k_\\mathrm{RBF}(x^{(i)}, x^{\\prime(i)}; \\ell^{(i)}).\n",
    "$$\n",
    "\n",
    "Here, $\\ell^{(i)}$ is the lengthscale associated with dimension $i$.\n",
    "Note that we are using a different lengthscale for each of the component kernels.\n",
    "Nevertheless, we can efficiently compute each of the component kernels in parallel using batching.\n",
    "First we define a RBFKernel object designed to compute a **batch of $d$ univariate kernels**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "90e52419-38cd-43e7-a599-bfb7abea89b8",
   "metadata": {},
   "outputs": [],
   "source": [
    "d = 3\n",
    "\n",
    "batch_univariate_rbf_kernel = gpytorch.kernels.RBFKernel(\n",
    "    batch_shape=torch.Size([d]),  # A batch of d...\n",
    "    ard_num_dims=1,  # ...univariate kernels\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba66b398-7429-408f-af23-d64ad6930df6",
   "metadata": {},
   "source": [
    "Including the `batch_shape` argument ensures that the `lengthscale` parameter of the `batch_univariate_rbf_kernel` is a `d x 1 x 1` tensor; i.e. each univariate kernel will have its own lengthscale. (We could instead have each univariate kernel share the same lengthscale by omitting the `batch_shape` argument.)\n",
    "\n",
    "To compute the univariate kernel matrices, we need to feed the appropriate dimensions of $\\boldsymbol X$ into each of the component kernels.\n",
    "We accomplish this by reshaping the `n x d` matrix representing $\\boldsymbol X$ into a batch of $d$ `n x 1` matrices\n",
    "(i.e. a `d x n x 1` tensor)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0edaca08-df1e-423d-b76c-e5f8f9fc5338",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 10\n",
    "\n",
    "X = torch.randn(n, d)  # Some random data in a n x d matrix\n",
    "batched_dimensions_of_X = X.mT.unsqueeze(-1)  # Now a d x n x 1 tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b40265f-a450-42fc-bcd4-f033750a8f7b",
   "metadata": {},
   "source": [
    "We then feed the batches of univariate data into the batched kernel object to get our batch of univariate kernel matrices:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e0988873-93fa-4016-880c-d9dff3344032",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 10, 10])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "univariate_rbf_covars = batch_univariate_rbf_kernel(batched_dimensions_of_X)\n",
    "univariate_rbf_covars.shape  # d x n x n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a61442cb-a575-44d9-ae97-cc396bc48a53",
   "metadata": {},
   "source": [
    "And finally, to get the multivariate kernel, we can compute the sum over the batch (i.e. the sum over the univariate kernels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "65cbeb00-4dde-4720-8d80-8a8b683caf8e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 10])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "additive_covar = univariate_rbf_covars.sum(dim=-3)  # Computes the sum over the batch dimension\n",
    "additive_covar.shape  # n x n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e52b2c1-35f2-4074-a5f5-36fd826d3b2e",
   "metadata": {},
   "source": [
    "On a small dataset, this approach is comparable to the naive approach described above. However, it will become much faster on a larger and more high dimensional dataset, especially on the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6a5d9b77-cd31-40b4-bc89-d913a601ea9b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<torch.utils.benchmark.utils.common.Measurement object at 0x7f35cc074640>\n",
      "naive_additive_kernel(X)\n",
      "  3.37 ms\n",
      "  1 measurement, 100 runs , 1 thread\n"
     ]
    }
   ],
   "source": [
    "d = 10\n",
    "n = 500\n",
    "device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n",
    "\n",
    "X = torch.randn(n, d, device=device)\n",
    "\n",
    "naive_additive_kernel = (\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[0]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[1]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[2]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[3]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[4]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[5]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[6]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[7]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[8]) +\n",
    "    gpytorch.kernels.RBFKernel(ard_num_dims=1, active_dims=[9])\n",
    ").to(device=device)\n",
    "\n",
    "with gpytorch.settings.lazily_evaluate_kernels(False):\n",
    "    print(benchmark.Timer(\n",
    "        stmt=\"naive_additive_kernel(X)\",\n",
    "        globals={\"naive_additive_kernel\": naive_additive_kernel, \"X\": X}\n",
    "    ).timeit(100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8ce95d7a-0419-4d15-b48c-892df4c7711e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<torch.utils.benchmark.utils.common.Measurement object at 0x7f35cc076ec0>\n",
      "batch_univariate_rbf_kernel(X.mT.unsqueeze(-1)).sum(dim=-3)\n",
      "  940.30 us\n",
      "  1 measurement, 100 runs , 1 thread\n"
     ]
    }
   ],
   "source": [
    "batch_univariate_rbf_kernel = gpytorch.kernels.RBFKernel(\n",
    "    batch_shape=torch.Size([d]), ard_num_dims=1,\n",
    ").to(device=device)\n",
    "with gpytorch.settings.lazily_evaluate_kernels(False):\n",
    "    print(benchmark.Timer(\n",
    "        stmt=\"batch_univariate_rbf_kernel(X.mT.unsqueeze(-1)).sum(dim=-3)\",\n",
    "        globals={\"batch_univariate_rbf_kernel\": batch_univariate_rbf_kernel, \"X\": X}\n",
    "    ).timeit(100))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa47ff89-dd55-44cf-b267-ac28ba0a9c27",
   "metadata": {},
   "source": [
    "### Full Example\n",
    "\n",
    "Putting it all together, a GP using this efficient additive kernel would look something like..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c32b0df1-1fa8-4b27-ab12-2d352ea80064",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AdditiveKernelGP(gpytorch.models.ExactGP):\n",
    "    def __init__(self, X_train, y_train, d):\n",
    "        likelihood = gpytorch.likelihoods.GaussianLikelihood()\n",
    "        super().__init__(X_train, y_train, likelihood)\n",
    "\n",
    "        self.mean_module = gpytorch.means.ConstantMean()\n",
    "        self.covar_module = gpytorch.kernels.ScaleKernel(\n",
    "            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([d]), ard_num_dims=1)\n",
    "        )\n",
    "\n",
    "    def forward(self, X):\n",
    "        mean = self.mean_module(X)\n",
    "        batched_dimensions_of_X = X.mT.unsqueeze(-1)  # Now a d x n x 1 tensor\n",
    "        covar = self.covar_module(batched_dimensions_of_X).sum(dim=-3)\n",
    "        return gpytorch.distributions.MultivariateNormal(mean, covar)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a9c2497-1200-44ed-b398-18a424976ea3",
   "metadata": {},
   "source": [
    "### Example #2: Efficient Products of Univariate Kernels\n",
    "\n",
    "As another example, we can consider a multivariate kernel defined as the product of univariate kernels, i.e.:\n",
    "\n",
    "$$\n",
    "    k_\\mathrm{RBF}(\\boldsymbol x, \\boldsymbol x'; \\boldsymbol \\ell) =  \\prod_{i=1}^d k_\\mathrm{RBF}(x^{(i)}, x^{\\prime(i)}; \\ell^{(i)}).\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "8122f229-c7ff-49df-a4ff-2cb3f9f05fe4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 10])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = 3\n",
    "n = 10\n",
    "\n",
    "batch_univariate_rbf_kernel = gpytorch.kernels.RBFKernel(\n",
    "    batch_shape=torch.Size([d]), ard_num_dims=1,\n",
    ")\n",
    "X = torch.randn(n, d)\n",
    "\n",
    "univariate_rbf_covars = batch_univariate_rbf_kernel(X.mT.unsqueeze(-1))\n",
    "with gpytorch.settings.lazily_evaluate_kernels(False):\n",
    "    prod_covar = univariate_rbf_covars.prod(dim=-3)\n",
    "prod_covar.shape  # n x n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68eaaa54-c481-45bb-aa77-fe79b6a6957d",
   "metadata": {},
   "source": [
    "This particular example is a bit silly, since the multivariate RBF kernel is exactly equivalent to the product of $d$ univariate RBF kernels,\n",
    "\n",
    "$$\n",
    "    k_\\mathrm{RBF}(\\boldsymbol x, \\boldsymbol x') =  \\prod_{i=1}^d k_\\mathrm{RBF}(x^{(i)}, x^{\\prime(i)}).\n",
    "$$\n",
    "\n",
    "However, this strategy can actually become advantageous when we approximate each of the univariate component kernels using a scalable $\\ll \\mathcal O(n^3)$ approximation for each of the univariate kernels.\n",
    "See [the tutorial on SKIP (structured kernel interpolation of products)](../02_Scalable_Exact_GPs/Scalable_Kernel_Interpolation_for_Products_CUDA.ipynb) for an example of exploiting product structure for scalability."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e83a742f-c24b-4cb2-be86-0c6e0cd67026",
   "metadata": {},
   "source": [
    "## Summing Higher Order Interactions Between Univariate Kernels (Additive Gaussian Processes)\n",
    "\n",
    "[Duvenaud et al. (2011)](https://arxiv.org/pdf/1112.4394) introduce \"Additive Gaussian Processes,\" which are GPs that additively compose interaction terms between univariate kernels.\n",
    "For example, with $d$-dimensional data and a max-degree of $3$ interaction terms, the corresponding kernel would be:\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "    k(\\boldsymbol x, \\boldsymbol x')\n",
    "    &= \\sum_{i=1}^d k_i(x^{(i)}, x^{\\prime(i)}; \\ell^{(i)}) \\\\\n",
    "    &+ \\sum_{i \\ne j} k_i(x^{(i)}, x^{\\prime(i)}; \\ell^{(i)}) k_j(x^{(j)}, x^{\\prime(j)}; \\ell^{(j)}) \\\\\n",
    "    &+ \\sum_{h \\ne i \\ne j} k_h(x^{(h)}, x^{\\prime(h)}; \\ell^{(h)}) k_i(x^{(i)}, x^{\\prime(i)}; \\ell^{(j)}) k_j(x^{(j)}, x^{\\prime(j)}; \\ell^{(j)})\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "Despite the summations having an exponential number of terms, this kernel can be computed in $\\mathcal O(d^2)$ time using the Newton-Girard formula.\n",
    "\n",
    "To compute this kernel in GPyTorch, we begin with a batch of the univariate covariance matrices (stored in a `d x n x n` Tensor or LinearOperator). We follow the same techniques as we used before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7ea7337c-6375-4287-8133-6117316e0791",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([4, 10, 10])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = 4\n",
    "n = 10\n",
    "\n",
    "batch_univariate_rbf_kernel = gpytorch.kernels.RBFKernel(\n",
    "    batch_shape=torch.Size([d]), ard_num_dims=1,\n",
    ")\n",
    "X = torch.randn(n, d)\n",
    "\n",
    "with gpytorch.settings.lazily_evaluate_kernels(False):\n",
    "    univariate_rbf_covars = batch_univariate_rbf_kernel(X.mT.unsqueeze(-1))\n",
    "univariate_rbf_covars.shape  # d x n x n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd7ba35a-87e3-4a86-acd7-de44184462cc",
   "metadata": {},
   "source": [
    "We then use the `gpytorch.utils.sum_interaction_terms` to compute and sum all of the higher-order interaction terms in $\\mathcal O(d^2)$ time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "7f6cb2da-46b6-4cd5-9e46-6e3ae49e8444",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10, 10])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "covar = gpytorch.utils.sum_interaction_terms(univariate_rbf_covars, max_degree=3, dim=-3)\n",
    "covar.shape  # n x n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1492c4cc-874b-4423-aab3-ea97501730bc",
   "metadata": {},
   "source": [
    "The full GP proposed by [Duvenaud et al. (2011)](https://arxiv.org/pdf/1112.4394) would then look like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f6dc72f7-d077-426c-9219-15b5cb371198",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AdditiveGP(gpytorch.models.ExactGP):\n",
    "    def __init__(self, X_train, y_train, d, max_degree):\n",
    "        likelihood = gpytorch.likelihoods.GaussianLikelihood()\n",
    "        super().__init__(X_train, y_train, likelihood)\n",
    "\n",
    "        self.mean_module = gpytorch.means.ConstantMean()\n",
    "        self.covar_module = gpytorch.kernels.ScaleKernel(\n",
    "            gpytorch.kernels.RBFKernel(batch_shape=torch.Size([d]), ard_num_dims=1)\n",
    "        )\n",
    "        self.max_degree = max_degree\n",
    "\n",
    "    def forward(self, X):\n",
    "        mean = self.mean_module(X)\n",
    "        batched_dimensions_of_X = X.mT.unsqueeze(-1)  # Now a d x n x 1 tensor\n",
    "        univariate_rbf_covars = self.covar_module(batched_dimensions_of_X)\n",
    "        covar = gpytorch.utils.sum_interaction_terms(\n",
    "            univariate_rbf_covars, max_degree=self.max_degree, dim=-3\n",
    "        )\n",
    "        return gpytorch.distributions.MultivariateNormal(mean, covar)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba613c79-92a4-41e7-92f7-4b6f598286bd",
   "metadata": {},
   "source": [
    "*(For those familiar with previous versions of GPyTorch, `sum_interaction_terms` replaces what was previously implemented by `NewtonGirardAdditiveKernel`.)*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
